We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces. The scale, quality, and diversity of object annotations far exceed those of prior datasets. A key difference setting apart HM3DSEM from other datasets is the use of texture information to annotate pixel-accurate object boundaries. We demonstrate the effectiveness of HM3DSEM dataset for the Object Goal Navigation task using different methods. Policies trained using HM3DSEM perform outperform those trained on prior datasets. Introduction of HM3DSEM in the Habitat ObjectNav Challenge lead to an increase in participation from 400 submissions in 2021 to 1022 submissions in 2022.

人类对象与铰接物体的相互作用在日常生活中很普遍。尽管单视图3D重建方面取得了很多进展，但从RGB视频中推断出一个铰接的3D对象模型仍然具有挑战性，显示一个人操纵对象的人。我们从RGB视频中划定了铰接的3D人体对象相互作用重建的任务，并对这项任务进行了五个方法家族的系统基准：3D平面估计，3D Cuboid估计，CAD模型拟合，隐式现场拟合以及自由 - 自由 - 形式网状配件。我们的实验表明，即使提供了有关观察到的对象的地面真相信息，所有方法也难以获得高精度结果。我们确定使任务具有挑战性的关键因素，并为这项具有挑战性的3D计算机视觉任务提出指示。短视频摘要https://www.youtube.com/watch?v=5talkbojzwc

建筑环境中许多物体的形状由他们与人体的关系决定：一个人将如何与这个对象进行互动？ 3D形状的现有数据驱动的生成模型产生合理的物体，但不会理由对人体的那些物体的关系。在本文中，我们学习了3D形状的身体感知生成模型。具体而言，我们培养椅子的生成型号，一种无处不在的形状类别，可以在给定的身体形状或坐姿姿势调节。身体形状调节的型号生产椅子，为具有给定体形的人舒适;姿势调节模型生产适应坐姿的椅子。要训​​练这些模型，我们定义了“坐姿匹配”度量标准和小说“坐姿舒适”度量。计算这些指标需要昂贵的优化将身体置于椅子上，这太慢被用作用于训练生成模型的损耗功能。因此，我们训练神经网络以有效地近似这些度量。我们使用我们的方法培训三个身体感知生成形状模型：基于结构的零件的发电机，点云发生器和隐式表面发生器。在所有情况下，我们的方法都生产适应其输出椅形状以输入人体规格的型号。

现实的3D室内场景数据集在计算机视觉，场景理解，自主导航和3D重建中启用了最近的最近进展。但是，现有数据集的规模，多样性和可定制性有限，并且扫描和注释更多的耗时和昂贵。幸运的是，组合者在我们方面：现有3D场景数据集有足够的个别房间，如果有一种方法可以将它们重新组合成新的布局。在本文中，我们提出了从现有3D房间生成新型3D平面图的任务。我们确定了这个问题的三个子任务：生成2D布局，检索兼容3D房间，以及3D房间的变形，以适应布局。然后，我们讨论解决问题的不同策略，设计两个代表性管道：一个使用可用的2D楼层计划，以指导3D房间的选择和变形;另一个学习检索一组兼容的3D房间，并将它们与新颖的布局相结合。我们设计一组指标，可评估所生成的结果与三个子任务中的每一个，并显示不同的方法在这些子任务上交易性能。最后，我们调查从生成的3D场景中受益的下游任务，并讨论选择最适合这些任务的需求的方法。

我们介绍了栖息地2.0（H2.0），这是一个模拟平台，用于培训交互式3D环境和复杂物理的场景中的虚拟机器人。我们为体现的AI堆栈 - 数据，仿真和基准任务做出了全面的贡献。具体来说，我们提出：（i）复制：一个由艺术家的，带注释的，可重新配置的3D公寓（匹配真实空间）与铰接对象（例如可以打开/关闭的橱柜和抽屉）； （ii）H2.0：一个高性能物理学的3D模拟器，其速度超过8-GPU节点上的每秒25,000个模拟步骤（实时850x实时），代表先前工作的100倍加速；和（iii）家庭助理基准（HAB）：一套辅助机器人（整理房屋，准备杂货，设置餐桌）的一套常见任务，以测试一系列移动操作功能。这些大规模的工程贡献使我们能够系统地比较长期结构化任务中的大规模加固学习（RL）和经典的感官平面操作（SPA）管道，并重点是对新对象，容器和布局的概括。 。我们发现（1）与层次结构相比，（1）平面RL政策在HAB上挣扎； （2）具有独立技能的层次结构遭受“交接问题”的困扰，（3）水疗管道比RL政策更脆。

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level library for end-toend development of embodied AI algorithms -defining tasks (e.g. navigation, instruction following, question answering), configuring, training, and benchmarking embodied agents.These large-scale engineering contributions enable us to answer scientific questions requiring experiments that were till now impracticable or 'merely' impractical. Specifically, in the context of point-goal navigation: (1) we revisit the comparison between learning and SLAM approaches from two recent works [20,16] and find evidence for the opposite conclusion -that learning outperforms SLAM if scaled to an order of magnitude more experience than previous investigations, and (2) we conduct the first cross-dataset generalization experiments {train, test} × {Matterport3D, Gibson} for multiple sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors generalize across datasets. We hope that our open-source platform and these findings will advance research in embodied AI.

Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available -current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.

This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG -a manually created largescale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task. The dataset, code and pretrained model will be available online upon acceptance.